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ABSTRACT 

This study investigated students' attitudes, engagement, satisfaction, and performance in Introduction to Environmental 
Science after it was transformed from a typical large lecture to a student-centered learning environment. The instructors of the 
course collaborated with the Office of Teaching, Learning & Technology and radically redesigned the course by adopting 
evidence-based pedagogical methods and instructional technology to offer a rich active learning environment. Students' 
engagement, satisfaction, and performance in the transformed course were compared to the students in the same course that 
was traditionally taught in a previous semester in order to measure the effect of the transformed course on student learning. 
Four self-reported surveys and a focus group interview were administered during the semester, and performance scores, prior 
learning data, and demographic information were collected at the end of the semester. The assessment results indicate that 
the students in the transformed course prepared for the class significantly more, engaged in the course significantly better, and 
were significantly more satisfied with the course than were the students in the traditional lecture-format course. Most of all, 
multiple linear regression indicates that the students in the transformed course earned, on average, about three-fourths of a 
letter grade higher on their final grade after controlling for cumulative grade point average. The students in the transformed 
course reported that online learning materials and frequent formative assessments using online quizzes were helpful in their 
learning, and they perceived the course was relevant to real-world applications that matter to their daily life. © 2016 National 
Association of Geoscience Teachers. [DOI: 10.5408/15-084.1] 
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INTRODUCTION 

Large lecture courses are a traditional teaching practice 
in most large institutions of higher education. They have 
historically provided an efficient way to deliver content 
information to a large number of students with the least 
amount of faculty resources. However, the traditional 
structure of a large lecture course adopts only one method 
of lecture, in which the instructor mainly delivers content 
knowledge. In the last a few decades, as assumptions about 
learning have evolved from the behaviorist perspective, in 
which learning is acquisition of information and routine 
skills (Greeno et al., 1996; Duffy and Orrill, 2001), to the 
constructivist view, much effort has been exerted in higher 
education to create environments for active learning. In 
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particular, such effort has been intensified in science, 
technology, engineering, and mathematics (STEM) fields. 
For example, in their reported titled "Engage to Excel," the 
President's Council of Advisers on Science and Technology 
called explicitly for widening the adoption of evidence-based 
teaching strategies in STEM to increase the number of 
college graduates with a STEM degree (Olson and Riordan, 
2012 ). 

The Office of Teaching, Learning & Technology at the 
University of Iowa initiated the Large Lecture Transforma¬ 
tion Project to increase the success of undergraduates. The 
primary goal of this project was to use technology to 
transform large lecture course delivery in order to provide 
increased student-centered learning. Most of all, through 
this project, sustainable models for alternative large lecture 
course delivery were developed to increase student engage¬ 
ment and satisfaction that led to greater student success. 
This project provides the necessary support structure to 
faculty members who wish to adopt new pedagogies and 
instructional technologies. A research study was conducted 
to examine the effectiveness of the project to transform 
traditional large lecture courses. As part of the Large Lecture 
Transformation Project, an Introduction to Environmental 
Science course was radically redesigned and transformed to 
offer a rich active learning environment. 

According to Grabinger (1996), new assumptions about 
learning, which evolved from constructivist theories, are 
moving from a passive view of learners who study concepts 
in abstract forms toward a paradigm in which learners 
construct knowledge through rich interaction in student- 
centered environments. In the constructivist view, learning is 
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a process of making sense of the world, which is in the 
activity of the learner (Duffy and Orrill, 2001). Therefore, the 
focus of learning has been changed from transmission of 
content to activities that involve learners. The constructivist 
approach of the optimal learning environment is to cultivate 
students' active construction of knowledge, which allows 
them to explore, discuss, and meaningfully construct 
concepts and relationships in contexts that involve real- 
world problems and projects that are relevant to the learners 
(Jonassen, 1999). As these new perspectives on learning 
have been broadly accepted by many educators, students' 
active engagement in the classroom has emerged as a critical 
factor in large lecture courses. In addition, much effort has 
been put into large lectures to create rich environments for 
active learning by leveraging a growing body of increasingly 
affordable instructional technologies. 

Many research studies have reported various changes of 
large lecture courses and their effectiveness on student 
engagement and performance. The scale of change has 
varied from adopting a single technology to reforming the 
course structure and assessment. One of the most commonly 
adopted technological tools in large lecture courses is an 
audience response system (or "clickers"), and many studies 
reported its positive results on student engagement and 
performance (e.g., Meltzer and Manivannan, 2002; Caldwell, 
2007; Mayer et al., 2009; Hoyt et al., 2010; Denker, 2013). 
Audience response systems promote not only student- 
instructor interaction but also student-student interaction 
and small group discussion with additional strategies. Bailey 
et al. (2012) indicated that adopting clickers with think-pair- 
share strategies promoted student-student interactions 
during lecture. 

Online learning components are often incorporated into 
large lecture courses to provide extensive learning materials 
to students and to allow students to build self-regulated 
learning skills (e.g., Letassy et al., 2008; McKenzie et al., 
2013). Multiple studies reported that video lectures improve 
students' initial learning (Brecht, 2012), keep students' 
attention, generate interest in science, and clarify under¬ 
standing (Eick and King, 2012). In particular, video lectures 
are more likely to benefit low-achieving than high-achieving 
students (Owston et al., 2011). Along with video lectures, 
online quizzes have been adopted in large lecture courses to 
prepare students for in-class activities. Commonly, quiz 
questions are based on assigned readings for each class and 
are due prior to the class. Because students can take online 
quizzes anywhere, they are not viewed as a valid assessing 
tool but rather as an incentive to have students read the 
materials prior to coming to the class (Aiken, 2005). 
Providing online self-tests without grading may also 
contribute to the substantial increase in student test scores 
(Bongey et al., 2005). 

Beyond these strategies, Persky and Pollack (2010) 
reported on the reform of a pharmacokinetics course by 
adding e-book and Web-based multimedia learning mod¬ 
ules to facilitate students' independent learning, which 
reduced face-to-face time. The majority of face-to-face 
lecture times were replaced with learning activities, includ¬ 
ing discussion, problem solving, and case studies, to 
encourage higher-order learning. Bailey et al. (2012) also 
reported on the reform of a large lecture course by including 
a process-oriented guided inquiry learning approach to 
create active learning environments in biochemistry. 


The effort to reform teaching practices by using 
interactive teaching pedagogies has been made specifically 
in the geosciences as well (e.g., Arthurs and Templeton, 
2009; Budd et al., 2013; Kennedy et al., 2013). Researchers 
observed 26 introductory geology courses at 11 institutions 
using the Reformed Teaching Observation Protocol. They 
found that 35% of them were student-centered classrooms 
with more active learning and 35% of them implemented 
some elements of active learning (Budd et al., 2013). This 
research also found that of observed courses, most large 
classes at research universities were traditional lecture- 
dominated classrooms with little student talk and minimal 
student activity beyond listening and note taking (Budd et 
al., 2013). The strategies that instructors adopted to create a 
student-centered learning environment in geoscience cours¬ 
es were varied, including coupled in-class small group 
activities and individual follow-up homework in a lecture- 
based environmental geology course (Arthurs and Temple¬ 
ton, 2009) and inquiry-based activities in a laboratory section 
in an introductory Earth Science course (Forcino, 2013). 

Most studies reported that redesigned large lecture 
courses showed relatively positive results compared to 
traditional lecture-format courses. Students achieved higher 
grades (Aitken, 2005; Walker et al., 2008), they preferred the 
redesigned course over the traditional lecture format, and 
their overall rating of the course was increased (e.g., Persky 
and Pollack, 2010). A recent meta-analysis of the impact of 
active learning on undergraduate student learning in STEM 
emphasized the importance of using evidence-based teach¬ 
ing strategies (Freeman et al., 2014). 

In contrast, a number of studies revealed no significant 
improvements in learner outcomes of redesigned large 
lecture courses even though a portion of the results indicated 
that overall, students valued their learning experience under 
the redesign (e.g., Kapp et al., 2011). Kahl's study (2008) 
pointed out a misalignment of interactive teaching methods 
and measures of student performance, in which the adopted 
assessment might not properly identify the degree to which 
students achieved the course goals. Rutledge (2008) exam¬ 
ined the effects of a diversified instructional approach on 
students' attitudes and reported that students in an 
instructor-centered large lecture course perceived it more 
effective than student-centered instruction in fostering 
knowledge of biological content, while student-centered 
instruction was perceived to be more effective in engaging 
students in the learning process and helping students 
construct their understanding. In addition, communication 
strategies have become an important factor in student 
success in redesigned courses, because students often do 
not see the purposes or values of activities on their own. 
Majerich and Schmuckler's study (2007) revealed that more 
positive student perceptions of the benefits of the adopted 
pedagogical methods resulted in greater content mastery for 
students in a modified course than for those in a traditional 
course. 

The current study investigated students' perceptions of 
the course and learning outcomes in the Introduction to 
Environmental Science course after it was transformed from 
a typical large lecture to a student-centered learning 
environment. In particular, students' attitudes, engagement, 
satisfaction, and performance in the transformed course 
were compared to those of students in the same course that 
was traditionally taught in order to measure the effect of the 
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FIGURE 1: Transformed process. 


transformed course on student learning. The same instruc¬ 
tors taught the course in the traditional lecture-centered 
course format a year before the transformed course was 
offered. 


SETTING 

University and Course Description 

The University of Iowa is a large public institution 
located in the Midwest. It is a research university and has 
more than 31,000 students. Introduction to Environmental 
Science is an introductory science course that fulfills the 
Natural Sciences General Education requirement at the 
University of Iowa. Prior to the course transformation 
discussed in this paper, this course was a traditional large 
lecture course: one instructor taught the first half of the 
course, and a second instructor taught the course for the 
second half of the semester. Class time was heavily focused 
on delivering content knowledge to about 200 to 250 
students in a large lecture hall. Lectures typically covered 
about two-thirds of a popular textbook at a pace of about 
one chapter per lecture, and instructors generally followed 
the order of content of the textbook. Students were assessed 
using two exams during the semester and a final compre¬ 
hensive exam. All exams consisted of multiple-choice 
questions. Writing is an important component of the 


General Education requirement at the University of Iowa. 
Prior to the course transformation, the writing component 
was a 250-word abstract associated with a service-learning 
experience. Due to the time required for assessing the 
writing assignments for a large population of students, no 
peer review or revision was incorporated into this assign¬ 
ment. 

Transformed Process 

The project team consisted of an instructional designer 
and an assessment researcher from the Office of Teaching, 
Learning & Technology and the course instructors and a 
senior teaching assistant from the Department of Earth & 
Environmental Sciences. The team had about 8 mo of 
preparation time before the transformed course was 
scheduled for implementation. The major motivations of 
instructors to participate in this project were to (1) increase 
students' engagement with course topics, (2) incorporate 
more meaningful writing projects, and (3) save students' cost 
by replacing a textbook with digital interactive learning 
materials. 

The process involved three stages: planning, design, and 
development (Fig. 1). First, at the planning stage, the 
instructors defined course goals for students in the course: 
(1) be able to critically evaluate popular media articles 
related to environmental science; (2) be able to locate and 
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TABLE I: Course structure and assessment comparison. 



Traditional Course 

Transformed Course 

Course Structure 

Two 75-min lectures per week 

Online learning 

One 50-min lecture by instructors 

TILE discussion 

Project session 

One 50-min lecture by guest lecturers 

Class Time 

150 min per week 

100 min per week 

Classrooms 

Lecture hall 

Lecture hall 

TILE classroom 

Learning Materials 

Textbook 

PowerPoint (PPT) slides by instmctors 

Online materials (text information, images, and videos) 

Lecture videos created by instructors 

PPT slides by instructors 

PPT slides by guest lecturers 

Assessment 

2 midterm tests (multiple-choice questions) 

1 final (multiple-choice questions) 

1 service learning project 

7 prelecture online quizzes (multiple-choice questions) 

7 module-cap online tests (multiple-choice questions) 

Clicker questions in the classroom (multiple-choice questions) 

2 individual writing assignments and peer assessments 

1 group writing assignments 

1 final (multiple-choice questions) 

1 service learning project 


interpret data relevant to environmental science; (3) 
understand the relationship between day-to-day life and 
environmental science, particularly in the context of sus¬ 
tainability; and (4) recognize the multifaceted nature of 
environmental science. The project team also outlined 
specific timelines for the tasks. 

At the design stage, the team made primary course 
design decisions after review of evidence-based practices 
reported in the literature. The selected revisions included 
plans to (1) increase online learning components for 
students to prepare prior to each class meeting, (2) provide 
authentic and hands-on learning experience in class, (3) 
provide frequent formative assessment opportunities and 
feedback to students, and (4) incorporate richer, multi¬ 
pronged assessment intended to measure students' achieve¬ 
ment of the newly defined course goals. 

At the design stage, the team decided to incorporate 
active learning in class and in the online learning compo¬ 
nents and to use quiz tools in the course site to provide 
frequent formative assessment opportunities and feedback. 
A student response system was also considered to provide 
formative assessment opportunities and increase students' 
engagement in class while allowing instructors to gauge 
students' understanding before the class activities. The team 
also decided to adopt a new technology, Calibrated Peer 
Review (CPR), to implement peer assessment for the writing 
projects. CPR is a Web-based application that was developed 
by the University of California at Los Angeles and enables 
frequent writing assignments with limited instructional 
resources (Walvoord et al., 2008). Incorporating major 
writing projects into the course would add enormous 
workload to the instructional staff, which was impossible 
with the current instructional staff capacity. Review of the 
literature on peer assessment indicated valuable learning 
benefits in both writing and critical-thinking skills from both 
receiving and providing feedback to peers' writing (e.g., 
Patchan and Schunn, 2015). Several studies reported that 
peer assessment using CPR was effective in improving 
student learning without adding substantial workload to 


instructors (e.g., Hartberg, 2008; Clase, 2010; Enders et al., 
2010; Likkel, 2012). 

In order to provide an authentic and hands-on learning 
experience in class, the team decided to adopt active learning 
classrooms. The University of Iowa offers multiple active 
learning classrooms (called TILE, for Transform, Interact, 
Learn, Engage) on campus. TILE classrooms are technology 
infused, which effectively facilitates group work. Each TILE 
classroom consists of round tables that seat nine students 
each, projectors and wall-mounted monitors that facilitate 
the sharing of information, and whiteboards for working out 
longer problems and for graphic communication. Each table 
contains three laptops, all connected to the projectors and 
wall-mounted monitors. Instructors were able to monitor 
the content displayed on student laptops and to push out the 
content of a selected laptop to all projectors and monitors to 
share with the class. Students could easily access all online 
resources through the laptops and share their work with the 
rest of the groups. The instructors decided to develop specific 
instructional activities that were designed for TILE classroom 
sessions. Guest lectures by local experts were also planned to 
provide examples and insights that could engage students in 
local environmental science issues. 

Next, in the development stage, the instructors devel¬ 
oped course materials (Table I). In place of a textbook, the 
instructors developed online content related to seven broad 
topics important to environmental science and society: 
science and controversy, dynamic earth systems, ecosystem 
ecology, water systems, food systems, carbon cycle, and 
climate change. These learning materials were developed 
from various open-source resources; the instructors also 
created lecture videos for certain modules. 

Two sets of quizzes, a preclass quiz and a module-cap 
test, were developed for each module. Preclass quizzes 
focused on the learning materials posted on the course Web 
site. The purpose of preclass quizzes was to have students 
prepare before they came to the first class of each module. 
Each module-cap test was designed for students to check 
their understanding of the whole topic at the end of the 
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module, including the posted learning materials and in-class 
activities. Only a part of each module-cap test was created in 
the development stage, and many items were dynamically 
created as each module was completed during the semester. 
The materials for two writing projects and peer assessment 
were developed as well. CPR requires writing prompts, a 
rubric, and three sample writings that guide students 
through the assessment process for their peers' writings. 

Instructional activities for the TILE classrooms focused 
on the hands-on activities in which students applied 
knowledge to the presented problems and created a type 
of solution (see the Supplemental TILE Plan; supplemental 
materials can be found online at http://dx.doi.org/10.5408/ 
15-084sl). Students' interactions and collaborative learning 
were emphasized in the design of the materials. All learning 
materials were organized with a consistent look in the 
course management system (Fig. 2) and included a list of 
specific learning objectives aligned to Bloom's taxonomy of 
learning (Bloom et al., 1956) for each module (Fig. 3). 
Bloom's taxonomy of learning suggests six cognitive 
outcomes that indicate different cognitive levels: knowledge, 
comprehension, application, analysis, synthesis, and evalu¬ 
ation (Anderson et al., 2001). 

The Office of Teaching, Learning & Technology 
purchased a license of CPR, installed it in the university 
network, and tested its capabilities for facilitating peer 
assessment. The main roles of the instructional designer 
during the transforming process were to facilitate the 
transforming process by arranging meetings and checking 
the working progress and to provide pedagogical and 
technological support to the instructors by recommending 
evidence-based pedagogical methods, providing feedback 
on developed instructional materials, coordinating technol¬ 
ogy and classrooms, and preparing technology documents 
for students. The Office of Teaching, Learning & Technology 
also provided technology training to the instructors. 

During the transforming process, the team focused on 
the consistent flow of all course components and learning 
materials tied to the main course goals that the instructors 
defined at the planning stage. The team also discussed 
evaluative criteria and determined the research design and 
tools to measure the effectiveness of innovations. The 
assessment researcher completed an institutional review 
board application and received approval before the begin¬ 
ning of the transformed course. 

Toward the end of the process, the team discussed the 
strategies for communicating with students effectively 
regarding course values, different course settings, and 
assessment and how to provide students with effective 
technology training. As a result, the instructor developed an 
introductory video providing a course overview and made it 
available on the course site (watch the Supplemental 
Introduction to Environmental Science video). The instruc¬ 
tors also planned the first class to communicate with 
students regarding the course overview and specific com¬ 
ponents. The team provided the teaching assistants with 
intensive training on how to support students in the 
transformed course so that they could effectively coach 
students with various tasks. 

Transformed Course 

The transformed course structure and class time were 
dramatically changed. The course design and assessment 


components are summarized in Table I. The original two 75- 
min lectures per week were replaced with a series of 2-week 
modules organized around one of the seven broad topics. 
Each module consisted of four 50-min sections: an instructor 
lecture, a project session, a TILE session, and a guest lecture. 
This structure reduced traditional instructor-student contact 
time by one-third. Students were required to study online 
learning materials, including lecture videos created by the 
instructors outside of class meetings, and to complete an 
online quiz to check their understanding prior to the first 
lecture of each module, as well as upon completion of the 
module. These supplemental activities were the substantial 
efforts for students, comprising the balance of the contact 
time. 

The instructors' lecture, attended by all students 
enrolled in the class, focused on deeper concepts and had 
students engage in exploration and meaningful discussion of 
the content they had mastered before coming to class. Even 
though this class was held in the traditional lecture hall, the 
instructors managed the seating area to facilitate group 
discussions. In the second 50-min session of each module, a 
subgroup of students (one-third of the students per group) 
met alternatively in the TILE session or project session to 
engage in the group discussion and hands-on activities in 
collaborative learning environments. In TILE sessions, 
students applied knowledge learned from online materials 
and lecture, produced group reports and products, and 
submitted them to the instructors at the end of the class for 
evaluation. During lecture and TILE sessions, instructors 
posted frequent clicker questions to assess student under¬ 
standing of core concepts, to initiate discussion, and to tailor 
their teaching. Instructors also identified the concepts that 
students missed most from the preclass quizzes and 
incorporated those concepts into the clicker questions. This 
helped students clarify difficult concepts. 

At the start of each module's second week, a local expert 
delivered a guest lecture focusing on research, an initiative, 
or a public policy issue germane to the module topic. The 
goal of the guest lecture was to expand student learning 
outside of the classroom and understand applications of the 
course material in the local community or region. The project 
session engaged students in writing through discussion of 
popular versus scientific writing styles, the peer review 
process, techniques for writing a review, and editing and 
revising under teaching assistants' and instructors' supervi¬ 
sion. The module-cap test at the end of each module covered 
all contents from online learning materials, instructor 
lecture, guest lecture, and TILE sessions, as well as some 
materials from previous modules as the semester progressed. 

In addition to frequent online quizzes, students had two 
writing and peer assessment assignments during the 
semester that were administered using the CPR application. 
The first writing assignment was to find an article from a 
popular online media source (e.g., CNN or Science Daily) 
and write a short essay that summarized reporting on a 
scientific study related to environmental science. After 
students submitted the essay, they calibrated as reviewers 
using three standards written by the instructors and 
evaluated three peers' writings based on a rubric that the 
instructors provided. After the peer review process, students 
reviewed the essay they had submitted at the beginning of 
the assignment using the same rubric. The second assign¬ 
ment was to write a short essay reviewing the scientific 
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ICON 


012:008:00A - Intro to Environmental Science 

Discussions Grades £3 Dropbox 


Table of Contents Modules Module 1. Science and Controversy 

_ Module 1. Science and Controversy * 


Module 1. Science and Controversy 

► 


View the learning objectives. 



Read the pre-lecture learning materials. 

By 01 28 



Watch "Common Misconceptions about Evolution" [8:04] 
By 01/28 



Download pdf 



Watch ”Example"[3:34] 

By 01 28 



Download pdf 


Take the quiz. 

By 01/28 at 11AM 

Take the module cap test. 

Open 02/04 at 5PM through 02/11 at 11AM 


FIGURE 2: Module presentation in the course management site. 
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□ i .be the ‘control ersv about the theory of evolution 


FIGURE 3: Document of learning objectives posted in the course management site. The document of learning 
objectives was presented in different colors keyed to indicate to learning contexts. The colors can be viewed only in 
the online version of the figure. 


article corresponding to the popular media article read in the 
first assignment and critiquing the how the scientific article 
was presented in the popular media article in light of the 
results and interpretations presented in the scientific paper. 
The same sequence—calibration, peer review, and self- 
assessment—was completed. 

All learning activities were presented in authentic 
contexts (Jonassen, 1999) so that students could see the 
connection between what they have learned in the course 
and what is happening in the real world and, in particular, in 
local contexts. Much effort was put into making meaningful 


relationships between students' day-to-day life and envi¬ 
ronmental science. The second week in each module, which 
was taught by an expert in the topic field, was closely related 
to current events or issues in the local area that might have 
had a direct impact on students' daily lives. 

The transformed course used technology to provide a 
rich active learning environment to students; at the same 
time, it reduced instructors' workload in facilitating activities. 
Clickers and technology in the TILE classroom greatly 
contributed to effective teaching and collaborative learning. 
In addition, the course management system was an effective 
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platform for communicating with students, accessing learn¬ 
ing materials, managing resources, and managing frequent 
formative assessments and feedback. The course manage¬ 
ment system provided various statistics that the instructors 
used to customize their teaching to fit students' needs. For 
example, the item statistics of each preclass quiz enabled 
instructors understand what concepts students were having 
difficulty with before starting the module. 

Writing assignments and peer assessments were possi¬ 
ble in this large enrollment class with the CPR system. Once 
the instructors created an assignment by entering writing 
prompts, rubrics, three sample writings, and answer keys 
and setting the grading criteria in the CPR system in 
advance, the CPR system then managed the students' 
activities. After students submitted their writings, the CPR 
system shuffled all writings and randomly assigned three 
students' writings to each student to review and evaluate. In 
reviews, students not only gave a numeric value but also 
provided detailed written feedback for each rubric item. 
After the reviews, students evaluated their own writing using 
the same rubric. Anonymity was guaranteed in the whole 
process. After the assignment, students accessed the results, 
including their performance on calibrations, reviews, and 
self-assessment, and evaluation results and detail feedback 
from all three reviewers. 


DATA COLLECTION 

Multiple data were collected to assess students' atti¬ 
tudes, engagement, satisfaction, and performance in the 
transformed course compared to the previous course that 
was traditionally taught. In addition, main course compo¬ 
nents were investigated related to students' perceptions of 
helpfulness to their learning. Student surveys were admin¬ 
istered four times during the semester: at the beginning and 
at end of the semester and two times in the middle of the 
semester. Participants' demographic information, prior 
learning (e.g., ACT scores and cumulative grade point 
average, or GPA), and course grades were collected at the 
end of the semester. In addition, one focus group interview 
was administered during the semester. A focus group 
interview and free-response comments in the self-reported 
surveys were used to assess students' perceptions of 
helpfulness of course components and overall perceptions 
of the transformed course in depth. After the entire interview 
was transcribed, the contents of the transcription and 
students' comments in the surveys were examined with 
respect to the main course components. 

Baseline Data From the Traditional Course 

The same instructors taught the course in the traditional 
lecture-centered course format a year before the transformed 
course was offered. In order to assess the effectiveness of the 
transformed course on students' attitudes, engagement, 
satisfaction, and performance, the baseline data from the 
same course were collected. The baseline data were from 191 
students who participated in the study. The assessment 
researcher collected those students' prior learning data, 
including cumulative GPA, demographic information, and 
performance scores, from the registrar's office and admin¬ 
istered a survey at the end of the semester. The survey was 
used to assess students' engagement level, satisfaction, 


interest, and preparedness; the same questions were used in 
the fourth survey in the transformed course. 

Survey Instruments 

Interest and Preparedness 

Five questions were asked to assess students' interest 
and preparedness for the class (see Supplemental Survey 
Questions). Each question had a 5-point response scale 
rating from 1 (never) to 5 (very often). 

Engagement 

The engagement measure contained three separate 
subscales: behavioral engagement, emotional engage¬ 
ment, and cognitive engagement. This measure consisted 
of 14 items with a 7-point response scale rating from 1 
(strongly disagree) to 7 (strongly agree). Behavioral and 
emotional engagement questions consisted of 10 ques¬ 
tions and were adopted from the questionnaire in Skinner 
et al.'s study (2009). Cognitive engagement questions 
consisted of 4 questions and were from Senko and Miles's 
study (2008). 

Satisfaction 

Four questions were created to assess students' overall 
satisfaction with the course. This measure included four 
Likert-scale items with a response scale ranging from 1 
(completely disagree) to 7 (completely agree). 

Course Objectives 

Four overall course objectives were presented to assess 
students' perceptions of helpfulness of the course to achieve 
them: (1) be able to critically evaluate popular media articles 
related to environmental science; (2) be able to locate and 
interpret data relevant to environmental science; (3) 
understand the relationship between day-to-day life and 
environmental science, particularly in the context of sus¬ 
tainability; and (4) recognize the multifaceted nature of 
environmental science. 

Course Components 

Nine course components were presented to assess 
students' perceptions of helpfulness of each component to 
achieve the course objectives (see Supplemental Survey 
Questions). 

Participants 

Data from 191 students (77% of the total enrollment) 
were collected in the traditional course for the baseline. Of 
the 191 students, male students were 53.4% of the 
participants, and freshmen were the largest group 
(38.7%). The next year, 132 students participated in the 
research study of the transformed course, representing 
85% of the total class enrollment. Of the 132 students, 
male students were 43.2% of the participants, and 
freshmen were the largest group, comprising 37.9% of 
the participants. White students were the largest group 
(75%) and the race/ethnicity information of nine students 
(6.8%) remained unknown (Table II). There were no 
significant differences in the demographic background of 
the students in terms of gender and academic years 
between the students in the traditional course and those in 
the transformed course. 
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TABLE II: Participants in traditional and transformed courses. 



Traditional 

Transformed 

Total Participants 

191 (77% of the total enrolled students) 

132 (85% of the total enrolled students) 

Gender 

Male: 53.4% 

Male: 43.2% 

Female: 46.6% 

Female: 56.8% 

Academic Year 

Freshmen: 38.7% 

Freshmen: 37.9% 

Sophomores: 26.5% 

Sophomores: 22% 

Juniors: 21.5% 

Juniors: 23.5% 

Seniors: 12.6% 

Seniors: 16.7% 

Other: 0.5% 


Ethnicity 


White: 75% 

African American: 2.3% 

Asians: 3.8% 

Hispanic or Latino: 6.1% 

International: 2.3% 

Multiracial: 3.8% 

Unknown: 6.8% 


Results 

Student Self-Reported Survey 

Student interest, preparedness, engagement, and satis¬ 
faction in both traditional and transformed courses were 
assessed by a student self-reported survey at the end of the 
semester. However, students' perceptions of helpfulness of 
the course to achieve the course goals and how helpful each 
course component was for achieving the course objectives 
were assessed only in the transformed course. In particular, 
students' perceptions of helpfulness of the course compo¬ 
nents were assessed two times in the middle of the semester. 

Five questions were asked to assess student interest in 
the course and preparedness. The results from an indepen¬ 
dent t -test indicated that students in the transformed course 
prepared for the lecture and were interested in the course 
significantly more than students in the traditional course 
(Table III). In particular, the effect size of completing all 
readings before the lecture was significantly large (d = 1.4). 
However, there was no significant difference regarding note 
taking during the lecture between two groups. 

In both courses, 14 questions were asked to assess 
student engagement at the end of the semester. The students 
in the transformed course reported a significantly higher 
engagement level than did the students in the traditional 
course. Among the three types of student engagement 
(behavioral, emotional, and cognitive engagement), behav¬ 
ioral and emotional engagement were significantly higher, 
with the p value at a 0.001 level, and cognitive engagement, 
with the p value at a 0.01 level, than the engagement levels 
in the traditional course (Table III). 

Four questions were asked to assess students' satisfac¬ 
tion at the end of the semester in both courses. The students 
in the transformed course indicated that they were more 
likely to take another course in environmental science than 
the students in the traditional course and were more 
interested in environmental science than they were before 
they took this course. The students in the transformed 
course reported higher overall satisfaction with the course 
than did the students in the traditional course (Table III). 


In the transformed course, we assessed how helpful the 
course activities were to the students for achieving the 
course goals. The instructors in the transformed course 
clearly defined four overall course goals and redesigned the 
course to facilitate students' achievement of these goals: (1) 
be able to critically evaluate popular media articles related to 
environmental science; (2) be able to locate and interpret 
data relevant to environmental science; (3) understand the 
relationship between day-to-day life and environmental 
science, particularly in the context of sustainability; and (4) 
recognize the multifaceted nature of environmental science. 
Students' perceptions of the effectiveness of the course to 
achieve those goals were positive. The means of students' 
responses on each goal ranged from 5.91 to 6.01 on a 7-point 
scale. More than 90% of students responded positively that 
the course was helpful to achieve the course goals. 

Two separate surveys (Surveys 2 and 3) asked students 
about how helpful each course component was for achieving 
the course objectives. Nine course components were 
presented to assess students' perceptions of helpfulness: 
online reading materials, online videos, lectures by the 
instructors, guest lectures, TILE activities, clicker questions, 
group projects, CPR writing assignments, and online 
quizzes. The CPR assignment was not included in Survey 
3, because there was no CPR assignment in that period. The 
means of students' responses on all components were higher 
than the midpoint. That is, students perceived all compo¬ 
nents to be relatively helpful. There were no significant 
differences between two times except in terms of clicker 
questions and group projects. Students responded that the 
clicker questions and group projects were less helpful in 
Survey 3 than they did in Survey 2. Among the nine 
components, students reported that online readings and 
video materials were the most helpful elements, whereas 
guest lectures, group projects, and CPR assignments were 
reported as less helpful. Surprisingly, students reported the 
online quizzes were helpful in their learning, along with 
online videos and readings (Fig. 4). 
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TABLE III: Results of f-test and descriptive statistics for students' interest in the course, preparedness, engagement, and 
satisfaction. 



Traditional 

Transformed 

95% Cl for 
Mean Difference 
(Upper, Lower) 

t 

df 

Mean 

SD 

n 

Mean 

SD 

n 

Interests and Preparedness 










How often did you complete all of the reading 
assignments before the lecture? 

2.06 

0.92 

141 

4.53 

0.84 

110 

2.70, 2.25 

21.84*** 

249 

How often did you bring your clicker with you to 
class? 

4.40 

1.12 

141 

4.87 

0.41 

110 

0.70, 0.24 

4.01*** 

249 

How often did you feel interested in the content of 
the lecture? 

2.50 

0.81 

141 

3.60 

1.03 

110 

1.30, 0.88 

9.5*** 

249 

How often did you talk about what you learned in 
the course with friends and family? 

2.27 

0.96 

141 

2.79 

1.28 

110 

0.80, 0.24 

3.69*** 

249 

How often did you take notes during the lecture? 

3.54 

1.44 

141 

3.28 

1.28 

109 

0.09, 0.60 

-1.45 

248 

Engagement (combined all 3 constructs ([14 questions]) 

4.58 

1.20 

140 

5.26 

0.99 

109 

0.96, 0.40 

4.75*** 

247 

Behavioral engagement 

4.69 

1.32 

140 

5.12 

0.94 

109 

1.04, 0.45 

4.99*** 

247 

Emotional engagement 

4.35 

1.43 

140 

5.22 

1.23 

109 

1.10, 0.42 

4.44*** 

247 

Cognitive engagement 

4.74 

1.44 

140 

5.26 

1.28 

109 

0.82, 0.13 

2.74** 

246 

Satisfaction 










Would take another course in environmental science 

4.26 

2.06 

140 

4.83 

1.91 

109 

1.06, 0.06 

2.20* 

247 

More interested in environmental science than I was 
before I took this class 

4.71 

1.88 

140 

5.24 

1.63 

109 

0.97, 0.08 

2.32* 

247 

Would recommend this course to my friends 

4.85 

1.80 

140 

5.60 

1.55 

109 

1.17, 0.32 

3.45*** 

247 

Overall satisfaction with this course 

4.87 

1.58 

136 

5.39 

1.42 

110 

0.90, 0.15 

2 77 ** 

244 


*p < 0.05. **p < 0.01. ***p < 0.001. 


Focus-Group Interview and Free-Response Comments in 
the Surveys 

In the transformed course, a focus-group interview was 
conducted to assess students' perceptions of each course 
component in depth. Three students participated in the 
focus-group interview. In the surveys, open-text fields were 
provided to collect students' comments about the course as 
well. Students' comments were organized by each course 
component, and their perceptions of each component were 
summarized. Students' quotes were directly extracted from 
the transcription of the focus-group interview and the 
comments in the surveys. 

Online Learning Materials and Quizzes 

Overall, students perceived the online learning mate¬ 
rials and quizzes as helpful. Students reported that the 
online materials were thorough and interesting, and they 
liked the flexibility of being able to read and watch videos 
on their schedule and in any location. Students felt that 
online quizzes were relatively difficult and detail oriented, 
but overall, students perceived them as helpful in their 
learning. 

"[I] liked everything available online to do at your own pace 
and when you have free time." (student 802003 from Survey 
4) 


"I really liked the biweekly quizzes/capstone tests. Breaking 
the class into modules was great and I really feel like I 
learned a lot.” (student 802047 from Survey 4) 

TILE Discussion 

Students perceived that the TILE discussion sessions 
were fun, were interactive, and allowed them to exchange 
ideas, and they believed that tasks and activities were closely 
connected to the real world. 

"J thought (TILE) was fun because it was very open 
ended.... It was interesting to see all of the other students, 
they came up with different ideas.... I like that ... they're 

fun and out of... non-traditional, and very interactive _ I 

liked the food one where we had to look up where food came 
from. I thought that was very real world... applicable... so 
like now when you're at the grocery store purchasing 
something you kind of like remember ... 'Oh I wonder 
where this came from.'" (student 1 from the focus group 
interview) 


"Having TILE activities to help give real world examples of 
the material we learned about in class."(student 802013 
from Survey 4) 


"Small quizzes kept me going. To be honest I feel like I am 
already prepared for my final exam." (student 802011 from 
Survey 4) 


CPR Assignments 

Students expressed mixed perceptions of the CPR 
assignments. The positive points were related to anonymity 
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FIGURE 4: Students' perception of helpfulness of the course components for achieving the course goals. 


and helpful guidelines in the process. Students did not 
know whose writings they were evaluating and who 
evaluated their writings. Students perceived that the rubrics 
were helpful because they clearly guided students in what 
to look for in peers' writings. Another positive point was 
the process of evaluating their own writing after reviewing 
three peers' writings because they could see their mistakes 
more easily. 

"[.I] paid more attention more about my writing, which is 
probably the whole purpose of it because you review your 

own _ It's very guided, you know. Peer review is usually 

pretty tough. I like that you don’t know whose paper it is, 
you don’t know who's writing it. And having 3 reviews was 

helpful _ The giddelines were easy to follow _ Like 

compare your writing to your peers so when ... I think 
that the last stage of the peer review that you have to review 
your own writing, and you kind of change your perspective 
writing after you've seen other people's and kind of you can 
see your mistakes better." (student 2 from the focus group 
interview) 

However, some students mentioned that learning the 
CPR system was confusing and that they did not like to be 
evaluated by peers. Not surprisingly, students seemed to 
need time to understand the CPR system. Several students 
expressed difficulty in the second survey, which they 
completed after the first CPR assignment. However, a few 


students mentioned a difficulty in the fourth survey, which 
they completed after the second CPR assignment. 

"The Calibrated Peer Review assignments are difficult to 
complete correctly. I do not feel editing papers has anything 
to do with Environmental Science." (student 802067 from 
Survey 2 after the first CPR assignment). 

The same student mentioned "CPR review from peers" 
as the most helpful aspect of this course. 

"CPR review from peers, gave me a better understanding on 
how to improve my writing." (student 802067from Survey 4 
after the second CPR assignment) 

Guest Lecture 

Students expressed mixed opinions about the helpful¬ 
ness of the guest lectures in their learning. A few students 
mentioned that the guest lecturers were hard to follow and 
some were too dry, while some students perceived they were 
helpful to understand real application in local areas and liked 
the connection between what they learned and what is 
happening in the real world. 

"I found the guest lecturers difficult to pay attention to and 
learn from. I had a hard time relating it to the class content 
and what I was supposed to be learning." (student 802046 
from Survey 4) 
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TABLE IV: Measures of learning outcomes. 



Traditional 

Transformed 

n 

Mean 

SD 

n 

Mean 

SD 

Cumulative GPA 

184 

2.89 

0.58 

131 

2.99 

0.62 

Final grade 

191 

2.49 

0.91 

129 

3.33 

0.88 


"I actually really liked the guest lectures.... We're learning 
from environmental scientists and it's interesting to learn 
from like paleontologists and biologist about the same sorts of 
things. So we hear different perspectives." (student 2 from 
the focus group interview) 

Overall Course 

Many students indicated positive perceptions of the 
course. First, students felt that their instructors were more 
approachable than the instructors in other large lecture 
courses. Second, providing different ways of learning, such 
as online learning, guest lectures, and TILE sessions, were 
viewed positively by many students. However, the various 
learning contexts presented challenges to some students. A 
few students expressed that it was hard to make connections 
among learned concepts from the different contexts of 
lecture, TILE discussion, and guest lecture. Furthermore, 
students felt that they needed self-regulated learning skills, 
planning, and better organization in order to succeed in this 
course. 

"I definitely appreciate that they're trying to go about a 
different approach to the class, it's not a traditional Monday, 
Wednesday, Friday lecture." (student 2 from the focus group 
interview) 


"I loved how the professors were our discussion teachers! It 
made it feel more personalized which was helpful to me 
because science is not my strong area. I felt more comfortable 
going up to these professors and asking questions than any of 
my other professors." (student 802124 from Survey 4) 

“The different components (lecture, TILE, discussion, lab) all 
coming together and reinforcing concepts learned in different 
ways." (student 802027 from Survey 4) 

"The biggest challenge was trying to incorporate lab, lecture, 
and TILE/project session. I felt they didn't flow well together, 
so it was a challenge to connect the material together." 
(student 802079 from Survey 4) 


Learning Outcomes 

Students' final exam results and course grades in both 
courses were collected to examine the effectiveness of 
transforming the course on students' learning outcomes. 
The mean score of students' final test in the transformed 
course was 64.27 (SD = 11.83), and the mean in the 
traditional course was 58.58 (SD = 10.26). Multiple linear 
regression analysis was used to test whether the course 
variable (traditional versus transformed) significantly pre¬ 
dicted students' final test performance. The results indicated 
that two predictors explained 47% of the variance [Ft 2 = 21.6, 
F( 2, 309) = 42.60, p < 0.001]. The course variable 
significantly predicted the final scores (P = 21.8, p < 
0.001), as did cumulative GPA (P = 38.9, p < 0.001). The 
results indicated that students in the transformed course 
achieved higher scores on the final test than did the students 
in the traditional course after controlling for their cumulative 
GPA. 

Likewise, students in the transformed course received 
higher grades than the students in the traditional course (F = 
95.44, p < 0.0001). The means of the final course grades were 
2.49 for the students in the traditional course and 3.33 for the 
students in the transformed course (Table IV). In the 
multiple linear regression analysis, students' cumulative 
GPA was controlled to assess the effect of the transformed 
course on students' final grade. The results indicate that the 
course variable was highly significant (P = 0.78, p < 0.0001): 
students in the transformed course earned, on average, 
about three-fourths of a letter grade higher on their final 
grade in the course after controlling for cumulative GPA 
(Table V). 


DISCUSSION 

The instructors defined clear course goals and rede¬ 
signed the course incorporating evidence-based pedagogy 
and instructional technology that had been previously 
supported by multiple research studies. Incorporating 
online learning elements seemed to have a positive impact 
not only on students' learning, with mastery of content 
knowledge, but also by developing self-regulated learning, 
which is one of the critical skills needed for success in 
academics (Schunk and Zimmerman, 1994). While students 
could access the learning materials anytime and anywhere, 
they needed to manage their own time effectively to learn 
the materials and complete the required assignments on 
time. One of the significant differences between the 
traditional and the transformed courses was students' 
preparedness for the course. The students in the trans¬ 
formed course reported that they read assigned readings 
significantly more often than the student in the traditional 
course. Furthermore, the students in the transformed 
course reported that online quizzes were one of the most 
helpful elements of the course. Many students expressed 


TABLE V: Parameter estimates for model predicting final grades. 


Parameter 

Estimate 

SE 

t Value 

Pr > \t\ 

Intercept 

0.43 

0.22 

1.95 

0.0524 

Cumulative GPA 

0.71 

0.07 

9.63 

<0.0001 

Transformed course (baseline = traditional course) 

0.78 

0.09 

8.87 

<0.0001 


F(2, 309) = 95.44, p < 0.0001. 
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that online quizzes helped them to keep on track during the 
semester and had them better prepared for the final exam. 
Frequent preclass and module-cap online quizzes seemed 
to foster self-regulated learning skills. In the transformed 
course, students regularly prepared for each class by 
studying the course materials, taking preclass quizzes, 
and reviewing what they learned at the end of the module. 
The module-cap online quiz included the materials from 
lecture, guest lecture, and TILE discussion, in which 
students could not perform well without attending and 
paying close attention to all classes. This preparation and 
the reviews might contribute to higher engagement in the 
transformed course compared to the course that was 
traditionally taught in 2013. 

Adding writing assignments and peer assessments 
using the CPR system seemed to contribute to not only 
expanding student learning outside of the classroom but 
also cultivating critical thinking skills of how to critique 
popular media articles related to science. As several 
students expressed, the CPR assignments provided oppor¬ 
tunities for students to identify mistakes in their writing 
and improve their writing skills. A few students, however, 
reported negative perceptions with the CPR assignments, 
such as feeling uncomfortable with being evaluated by 
peers, difficulty to understand the CPR system to complete 
assignments correctly, and not fully understanding of the 
purpose of writing assignments in a science course. The 
nature of students' resistance to peer assessment in higher 
education has been explored by several research studies 
(e.g., Kaufman and Schunn, 2011). This resistance has been 
documented despite the findings that peer assessment has 
improved student writing and critical thinking skills (e.g., 
Topping, 1998). However, that resistance can be alleviated 
when students have positive learning experiences in the 
process of giving and receiving helpful feedback. Instruc¬ 
tors' thoughtful design of writing assignments and rubrics 
are essential for guiding students effectively, and proper 
training and communicational strategies are required in 
advance. Students need to understand the values and 
purpose of the activities to engage in the tasks meaningfully 
and gain the positive learning experiences as instructors 
intend. 

Incorporating TILE discussion seemed to have a positive 
impact on students' engagement. Unlike a typical large 
lecture hall, the active learning classroom provides an 
informal, relaxed, and egalitarian classroom environment, 
and it fosters learning alliances between students and 
instructors and among students (Baepler and Walker, 
2014). Several students mentioned that their instructors 
were more approachable than other instructors in typical 
large lecture courses that they had taken. Informal and 
relaxed classroom environments might contribute to creating 
positive relationships between students and instructors. In 
addition, the physical layout of the TILE classroom facilitated 
collaborative learning effectively, and the technology in the 
room might increase the productivity and creativeness of 
group work. 

Jonassen (1999) specified that the optimal learning 
environment cultivates students' active construction, allow¬ 
ing students to explore, discuss, and meaningfully construct 
concepts and relationships in contexts that involve real- 
world problems and projects that are relevant to the learners. 
Many students in the transformed course appreciated the 


different ways of learning and enjoyed engaging in multiple 
tasks that were connected to real-world applications. 
Presenting the course contents in various contexts and 
having students engage in hands-on projects in collaborative 
learning environments seemed to increase students' interest 
and engagement. 

Limitations and Future Work 

While assessment results showed a significant increase 
in student engagement and learning outcomes, two recur¬ 
ring negative comments from the students should be 
recognized. First, connectivity of multiple learning contexts 
was mentioned as an issue. Although the instructors tried to 
create good flow among different learning contexts—large 
class sessions, TILE sessions, guest lectures, and project 
sessions—some students could not fully understand the core 
concepts and main objectives of those different contexts and 
seemed to be confused. Many students are accustomed to a 
typical large lecture format and might expect the same 
learning environments throughout the semester. Therefore, 
multiple learning contexts might sometimes disrupt their 
flow of learning. Additional communication strategies, 
including visual elements, could be developed to show 
how different learning contexts come together to foster the 
learning process. 

Second, the guest lectures should be reconsidered. 
While many students perceived guest lectures as helpful 
for understanding how their learning applied to real-world 
settings, some students viewed them as too dry and hard to 
understand. The course instructors cannot control the 
content and delivery of guest lectures. All guest lectures 
should be reviewed, and those that fit broad student 
interests and students' level of understanding should be 
incorporated into the class. 

Lastly, among the course components, students re¬ 
sponded that the clicker questions and group projects were 
less helpful in Survey 3 than in Survey 2. The third survey 
was conducted after completing Modules 4 and 5, whereas 
the second survey was given after completing the first three 
modules. The clicker questions and group projects in 
Modules 4 and 5 should be reexamined and revised, if 
necessary, to improve student learning. 

Essentials for Course Transformation 

Redesigning a course, in particular, a large enrollment 
course, is a daunting task to instructors, who lack time, 
resources, and a background in educational design. How¬ 
ever, collaboration with the Office of Teaching, Learning & 
Technology made this transformation successful. The team 
started the process 8 mo before offering the redesigned 
course, which was the appropriate time to plan, design the 
course components, and develop course materials. The clear 
course goals that instructors defined in the planning stage 
were critical in the design and development stages. The 
course goals were developed in terms of learning outcomes, 
which students should have achieved when they completed 
the course. All design decisions were made and materials 
were developed to achieve the course goals. Pedagogical and 
instructional technology support from the instructional 
designer assisted instructors not only with the decision¬ 
making process but also with the coordination of multiple 
instructional technologies in the process so that instructors 
could focus on the learning materials and instructional 
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activities that would matter most in student learning. Having 
sufficient time to redesign the course was essential, in 
particular, when new pedagogical methods and instructional 
technology were implemented. Instructors needed to fully 
understand the advantages of newly adopted instructional 
technology and to be aware of potential issues that might 
occur during the semester. To be effective in implementing 
new course activities with instructional technology, instruc¬ 
tors need to address those potential issues to facilitate their 
students' learning. The key to success in this transformation 
was the continued support of all team members through all 
project stages, including implementation and discussion of 
outcomes. 


CONCLUSIONS 

Today's complex world demands that employees be 
creative and flexible problem solvers who can work 
effectively in groups rather than simply knowing how to 
use tools and knowledge in a single domain (Grabinger, 
1996). In order to be a creative and flexible problem solver 
and effective collaborator, students must learn to think 
critically and apply knowledge to multiple contexts in 
collaborative learning environments. 

Redesigning a course to create a rich active learning 
environment is not a simple task for instructors who teach a 
large number of students. However, the instructors of the 
Introduction to Environmental Science course collaborated 
with the team of the Office of Teaching, Learning & 
Technology and redesigned the course by adopting evi¬ 
dence-based pedagogy and technology to provide a rich 
active learning environment. This collaborative work signif¬ 
icantly increased student engagement and learning out¬ 
comes compared with the same course that was traditionally 
taught by the same instructors in a previous semester. This 
study suggests that clearly defined course goals and 
measurable learning outcomes will initiate the process of 
redesigning a course to encourage meaningful student 
engagement, thereby leading to better student success. 
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